class: center, middle, inverse, title-slide # Lecture 19 ## Simple Linear Regression ### Psych 10 C ### University of California, Irvine ### 05/13/2022 --- ## Simple linear regression - We will start with another example using simple linear regression. -- - We are interested in studying how the time that it takes participants to identify a geometrical object changes as a function of the degrees by which the object is rotated. -- - For example, in a single trial of the experiment we present a participant with a geometric figure (left), and then, after a a fixed period of time has elapsed (say 10 seconds), we present the participant another figure (right). The objective of the participant is to respond whether the figure is the same or not. .pull-left[ .center[] ] .pull-right[ .center[] ] --- ## Example: Mental rotation - We presented participants with 6 figures rotated 10, 45, 90, 130, 160 or 180 degrees respectively, and recorded the time it took them to identify if the second image was the same figure. -- - The age of participants in the experiment ranged from 7 to 21 years. -- - The data from the looks like this: --
--- ## Example: Mental rotation - Response times are saved in milliseconds (ms), thus the large numbers. -- - For now we will assume that the age of the participant has no effect on response times. -- - This is how we usually approach data analysis with linear models. Remember that the focus of our research question is on the effect of the **angle of rotation** on response time, not the **age**. -- - Because we are interested in the effect of angle of rotation we will start with a simple linear regression using only that predictor. --- ## Null model - The first model is the Null, which assumes that there is no effect of angle of rotation on the time it takes participants to identify if the figure is the same but has been rotated (response time). -- - Remember the model is formalized as: `$$y_i \sim \text{Normal}(\beta_0, \sigma_0^2)$$` -- - Were `\(y_i\)` represents the *i-th* response time in the experiment, `\(\beta_0\)` is the expected response time according to the Null model, and `\(\sigma_0^2\)` is the variance of response times according to the Null model. -- - As we have mentioned before, the best guess for the expected value of a Normal distribution (when there is only one parameter) is the average of all response times in the experiment. -- - Therefore, `$$\hat{\beta}_0 = \bar{y} = \frac{\sum_{i=1}^{n} y_i}{n}$$` --- ## Regression model - The second model is the simple linear regression. This model assumes that the expected response time changes as a linear function of the degrees by which the figure has been rotated. -- - In other words, the model assumes that the **angle of rotation** is a good predictor of response times. -- - The model is formalized as follows: `$$y_i \sim \text{Normal}(\beta_0+\beta_1x_i, \sigma_1^2)$$` -- - Were `\(y_i\)` represents the *i-th* response time. -- - `\(\beta_0\)` represents the response time to a figure that has been rotated by 0 degrees. --- ## Regression model - `\(\beta_1\)` represents the change in response time for a 1 degree increase in the rotation of the figure. For example, the change in response time to a figure that has been rotated 10 degrees in comparison to one that has been rotated 11 degrees. -- - `\(x_i\)` represents the angle of rotation of the *i-th* figure presented to the participant. -- - Finally, `\(\sigma_1^2\)` represents the variability of response times with respect to their expectation `\(\beta_0 + \beta_1x_i\)`. -- - Notice that this model assumes that the expected response time will be different depending on the angle of rotation of the figure. In other words, we expect response times to be one value for figures rotated 10 degrees and a different one when the figure is rotated 10.1 degrees. -- - In other words, our predictions about the expectation of response times are a continuous variable that follows a straight line! --- ## Regression model - The prediction of the linear model for a given angle of rotation `\(x_i\)` is: `$$\hat{\beta}_0 + \hat{\beta}_1x_i$$` -- - We can obtain the estimators `\(\hat{\beta}_0\)` and `\(\hat{\beta}_1\)` using the **`lm()`** function in R. -- - Remember that the function in R takes two arguments "**`formula =`**" which has to be written as: `$$\text{dependent-variable} \sim \text{independent-variable}$$` -- - The second argument is "**`data =`**" which is the name of the object that contains our observations. --- ## Predictions: Null model - Let's start by adding the predictions of the Null model to the data and calculating the squared error by observation. -- ```r # Total sample size n_total <- nrow(mental_rotation) # Prediction of the Null model null <- mental_rotation %>% summarise("pred" = mean(response_time)) %>% pull(pred) # Add prediction to data and calculate error mental_rotation <- mental_rotation %>% mutate("prediction_null" = null, "error_null" = (response_time - prediction_null)^2) # Calculate the Sum of Squared Error of the Null model sse_null <- sum(mental_rotation$error_null) # Calculate the Mean Squared Error of the Null model mse_null <- 1/n_total * sse_null # Calculate the BIC for the Null model bic_null <- n_total * log(mse_null) + 1 * log(n_total) ``` --- ## Predictions: Linear regression - Now we can add the prediction and error of the "rotation" model. -- ```r # Obtain estimators for beta0 and beta1 betas <- lm(formula = response_time ~ angle, data = mental_rotation)$coef # Add prediction and error of the linear model mental_rotation <- mental_rotation %>% mutate("prediction_rotation" = betas[1] + betas[2] * angle, "error_rotation" = (response_time - prediction_rotation)^2, "epsilon_rotation" = (response_time - prediction_rotation)) # Calculate the SSE of the linear model sse_rotation <- sum(mental_rotation$error_rotation) # Calculate the MSE of the linear model mse_rotation <- 1/n_total * sse_rotation # Calculate the variance accounted for by the linear model r2_rotation <- (sse_null - sse_rotation)/sse_null # Calculate the BIC of the linear model bic_rotation <- n_total * log(mse_rotation) + 2 * log(n_total) ``` --- ## Results - Now that we have calculated all the values that we need, we can summarize them on a table in order to compare our two models and answer our research question. -- | Model | Parameters | MSE | `\(R^2\)` | BIC | |-------|:----------:|:---------------------:|:-----:|:--------------------:| | Null | 1 | `\(8.8083\times 10^{5}\)` | NA | 4112| | Angle of Rotation | 2 | `\(4.2866\times 10^{4}\)` | 0.95 | 3211| -- - The results indicate that the angle of rotation of a figure is a good predictor of the time it takes participants to identify if a figure is the same as the one they saw at the start of the trial that has been rotated. -- - We can also interpret the estimated values of the parameters in the model. --- ## Interpretation of model parameters - When we interpret the parameters of a linear model we usually write something like this: -- - The estimated value of the intercept was 453, which means that it takes participants 453 milliseconds on **average** to identify a figure that **was not rotated** (rotated 0 degrees). -- - The estimated value of the slope was 15, this suggests that it takes participants on **average** 15 milliseconds **more** to identify a figure **for each additional degree** that the figure has been rotated by. -- - Notice that when we interpret the parameters we always have to say "**on average**", this is because the model's predictions are about the expected response times, which means that there is error around those predicted values. -- - Furthermore, the interpretation has to be made by making reference to the variables on the study, for example, "**expected response time in milliseconds**" or "**for each additional degree of rotation**". This is because each estimate is associated with a different variable in the study. --- ## Evaluating the linear model - In the code of the linear model we added an additional calculation called **`epsilon_rotation`**, this variable is just the difference between observation and model prediction without squaring the value. -- - As we have said before, if we add the difference between observation and model prediction that difference will be equal to 0 (by definition). This is true regardless of the situation, however, we can use those differences to evaluate our model in a more specific way. -- - In general we want those differences to have two properties: -- 1. The difference between observations and predictions follows approximately a Normal distribution centered at 0. This means that if we make a histogram of the difference, the histogram should be centered at 0 and be symmetric. -- 2. There are no patterns, in the difference between observation and prediction of the model. This means that regardless of how we define our *x-axis* on a graph, the difference between observation and prediction should look like white noise (there are no obvious patterns). --- count: false ### Histogram of the difference between observation and prediction .panel1-epsilon-hist-auto[ ```r * ggplot(data = mental_rotation) ``` ] .panel2-epsilon-hist-auto[ <!-- --> ] --- count: false ### Histogram of the difference between observation and prediction .panel1-epsilon-hist-auto[ ```r ggplot(data = mental_rotation) + * aes(x = epsilon_rotation) ``` ] .panel2-epsilon-hist-auto[ <!-- --> ] --- count: false ### Histogram of the difference between observation and prediction .panel1-epsilon-hist-auto[ ```r ggplot(data = mental_rotation) + aes(x = epsilon_rotation) + * geom_histogram(color = "#E72F52", fill = "#E72F52", binwidth = 40, * aes(y =..density..), ) ``` ] .panel2-epsilon-hist-auto[ <!-- --> ] --- count: false ### Histogram of the difference between observation and prediction .panel1-epsilon-hist-auto[ ```r ggplot(data = mental_rotation) + aes(x = epsilon_rotation) + geom_histogram(color = "#E72F52", fill = "#E72F52", binwidth = 40, aes(y =..density..), ) + * theme_classic() ``` ] .panel2-epsilon-hist-auto[ <!-- --> ] --- count: false ### Histogram of the difference between observation and prediction .panel1-epsilon-hist-auto[ ```r ggplot(data = mental_rotation) + aes(x = epsilon_rotation) + geom_histogram(color = "#E72F52", fill = "#E72F52", binwidth = 40, aes(y =..density..), ) + theme_classic() + * xlab("Response time - Prediction") ``` ] .panel2-epsilon-hist-auto[ <!-- --> ] --- count: false ### Histogram of the difference between observation and prediction .panel1-epsilon-hist-auto[ ```r ggplot(data = mental_rotation) + aes(x = epsilon_rotation) + geom_histogram(color = "#E72F52", fill = "#E72F52", binwidth = 40, aes(y =..density..), ) + theme_classic() + xlab("Response time - Prediction") + * ylab("") ``` ] .panel2-epsilon-hist-auto[ <!-- --> ] --- count: false ### Histogram of the difference between observation and prediction .panel1-epsilon-hist-auto[ ```r ggplot(data = mental_rotation) + aes(x = epsilon_rotation) + geom_histogram(color = "#E72F52", fill = "#E72F52", binwidth = 40, aes(y =..density..), ) + theme_classic() + xlab("Response time - Prediction") + ylab("") + * theme(axis.title.x = element_text(size = 20), * axis.title.y = element_text(size = 20)) ``` ] .panel2-epsilon-hist-auto[ <!-- --> ] --- count: false ### Histogram of the difference between observation and prediction .panel1-epsilon-hist-auto[ ```r ggplot(data = mental_rotation) + aes(x = epsilon_rotation) + geom_histogram(color = "#E72F52", fill = "#E72F52", binwidth = 40, aes(y =..density..), ) + theme_classic() + xlab("Response time - Prediction") + ylab("") + theme(axis.title.x = element_text(size = 20), axis.title.y = element_text(size = 20)) + * stat_function(fun = dnorm, * args = list(mean = 0, sd = sqrt(mse_rotation))) ``` ] .panel2-epsilon-hist-auto[ <!-- --> ] <style> .panel1-epsilon-hist-auto { color: black; width: 38.6060606060606%; hight: 32%; float: left; padding-left: 1%; font-size: 80% } .panel2-epsilon-hist-auto { color: black; width: 59.3939393939394%; hight: 32%; float: left; padding-left: 1%; font-size: 80% } .panel3-epsilon-hist-auto { color: black; width: NA%; hight: 33%; float: left; padding-left: 1%; font-size: 80% } </style> --- count: false ### Difference between observation and prediction .panel1-epsilon-index-auto[ ```r * ggplot(data = mental_rotation) ``` ] .panel2-epsilon-index-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction .panel1-epsilon-index-auto[ ```r ggplot(data = mental_rotation) + * aes(x = as.numeric(rownames(mental_rotation))) ``` ] .panel2-epsilon-index-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction .panel1-epsilon-index-auto[ ```r ggplot(data = mental_rotation) + aes(x = as.numeric(rownames(mental_rotation))) + * aes(y = epsilon_rotation) ``` ] .panel2-epsilon-index-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction .panel1-epsilon-index-auto[ ```r ggplot(data = mental_rotation) + aes(x = as.numeric(rownames(mental_rotation))) + aes(y = epsilon_rotation) + * geom_point(color = "#E72F52", size = 3) ``` ] .panel2-epsilon-index-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction .panel1-epsilon-index-auto[ ```r ggplot(data = mental_rotation) + aes(x = as.numeric(rownames(mental_rotation))) + aes(y = epsilon_rotation) + geom_point(color = "#E72F52", size = 3) + * theme_classic() ``` ] .panel2-epsilon-index-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction .panel1-epsilon-index-auto[ ```r ggplot(data = mental_rotation) + aes(x = as.numeric(rownames(mental_rotation))) + aes(y = epsilon_rotation) + geom_point(color = "#E72F52", size = 3) + theme_classic() + * xlab("Observation number") ``` ] .panel2-epsilon-index-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction .panel1-epsilon-index-auto[ ```r ggplot(data = mental_rotation) + aes(x = as.numeric(rownames(mental_rotation))) + aes(y = epsilon_rotation) + geom_point(color = "#E72F52", size = 3) + theme_classic() + xlab("Observation number") + * ylab("Response time - Prediction") ``` ] .panel2-epsilon-index-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction .panel1-epsilon-index-auto[ ```r ggplot(data = mental_rotation) + aes(x = as.numeric(rownames(mental_rotation))) + aes(y = epsilon_rotation) + geom_point(color = "#E72F52", size = 3) + theme_classic() + xlab("Observation number") + ylab("Response time - Prediction") + * theme(axis.title.x = element_text(size = 20), * axis.title.y = element_text(size = 20)) ``` ] .panel2-epsilon-index-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction .panel1-epsilon-index-auto[ ```r ggplot(data = mental_rotation) + aes(x = as.numeric(rownames(mental_rotation))) + aes(y = epsilon_rotation) + geom_point(color = "#E72F52", size = 3) + theme_classic() + xlab("Observation number") + ylab("Response time - Prediction") + theme(axis.title.x = element_text(size = 20), axis.title.y = element_text(size = 20)) + * geom_smooth(se = FALSE, size = 2, color = "#0d95d0") ``` ] .panel2-epsilon-index-auto[ <!-- --> ] <style> .panel1-epsilon-index-auto { color: black; width: 38.6060606060606%; hight: 32%; float: left; padding-left: 1%; font-size: 80% } .panel2-epsilon-index-auto { color: black; width: 59.3939393939394%; hight: 32%; float: left; padding-left: 1%; font-size: 80% } .panel3-epsilon-index-auto { color: black; width: NA%; hight: 33%; float: left; padding-left: 1%; font-size: 80% } </style> --- count: false ### Difference between observation and prediction by age .panel1-epsilon-age-auto[ ```r * ggplot(data = mental_rotation) ``` ] .panel2-epsilon-age-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction by age .panel1-epsilon-age-auto[ ```r ggplot(data = mental_rotation) + * aes(x = age) ``` ] .panel2-epsilon-age-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction by age .panel1-epsilon-age-auto[ ```r ggplot(data = mental_rotation) + aes(x = age) + * aes(y = epsilon_rotation) ``` ] .panel2-epsilon-age-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction by age .panel1-epsilon-age-auto[ ```r ggplot(data = mental_rotation) + aes(x = age) + aes(y = epsilon_rotation) + * geom_point(color = "#E72F52", size = 3) ``` ] .panel2-epsilon-age-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction by age .panel1-epsilon-age-auto[ ```r ggplot(data = mental_rotation) + aes(x = age) + aes(y = epsilon_rotation) + geom_point(color = "#E72F52", size = 3) + * theme_classic() ``` ] .panel2-epsilon-age-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction by age .panel1-epsilon-age-auto[ ```r ggplot(data = mental_rotation) + aes(x = age) + aes(y = epsilon_rotation) + geom_point(color = "#E72F52", size = 3) + theme_classic() + * xlab("Participant age") ``` ] .panel2-epsilon-age-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction by age .panel1-epsilon-age-auto[ ```r ggplot(data = mental_rotation) + aes(x = age) + aes(y = epsilon_rotation) + geom_point(color = "#E72F52", size = 3) + theme_classic() + xlab("Participant age") + * ylab("Response time - Prediction") ``` ] .panel2-epsilon-age-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction by age .panel1-epsilon-age-auto[ ```r ggplot(data = mental_rotation) + aes(x = age) + aes(y = epsilon_rotation) + geom_point(color = "#E72F52", size = 3) + theme_classic() + xlab("Participant age") + ylab("Response time - Prediction") + * theme(axis.title.x = element_text(size = 20), * axis.title.y = element_text(size = 20)) ``` ] .panel2-epsilon-age-auto[ <!-- --> ] --- count: false ### Difference between observation and prediction by age .panel1-epsilon-age-auto[ ```r ggplot(data = mental_rotation) + aes(x = age) + aes(y = epsilon_rotation) + geom_point(color = "#E72F52", size = 3) + theme_classic() + xlab("Participant age") + ylab("Response time - Prediction") + theme(axis.title.x = element_text(size = 20), axis.title.y = element_text(size = 20)) + * geom_smooth(se = FALSE, size = 2, color = "#0d95d0") ``` ] .panel2-epsilon-age-auto[ <!-- --> ] <style> .panel1-epsilon-age-auto { color: black; width: 38.6060606060606%; hight: 32%; float: left; padding-left: 1%; font-size: 80% } .panel2-epsilon-age-auto { color: black; width: 59.3939393939394%; hight: 32%; float: left; padding-left: 1%; font-size: 80% } .panel3-epsilon-age-auto { color: black; width: NA%; hight: 33%; float: left; padding-left: 1%; font-size: 80% } </style> --- ## Evaluating the linear model - The last graph showed us that there was a trend on the difference between the response time of participants and the predictions of the model. -- - The blue line showed us that, on average, the error of the model decreased with the age of the participants. -- - Whenever, we see a trend on the difference between observation and prediction this suggests that there is an important variable that we are not taking into account. -- - In this case, the **response time** of participants in the task seems to be associated not only with the **angle of rotation** of a figure but also to the **age** of the participant. -- - In order to be able to take this second variable into account we need to extend our linear model by considering more than one independent variable. --- ## Multiple linear regression - The main idea of a multiple linear regression is that we can add more variables to a linear model in order to improve our predictions. -- - In a multiple linear regression model we can add more than one predictor (independent variable) which will be associated to a new parameter. -- - In our example we want to add to the model the **age** of participants as a new predictor of response time. -- - The model is formalized this way: `$$y_i\sim\text{Normal}(\beta_0 + \beta_1x_{i1} + \beta_2x_{i2}, \sigma_m^2)$$` -- - Were `\(y_i\)` represents the response time of the *i-th* participant, `\(x_{i1}\)` represents the **angle of rotation** associated with the *i-th* observation and `\(x_{i2}\)` represents the **age** of the participant associated with the *i-th* observation. --- ## Multiple linear regression - This new model now has 3 parameters associated with its predictions, `\(\beta_0\)` or the **intercept**, `\(\beta_1\)` or the **slope** associated with a change in the angle of rotation of a figure, and `\(\beta_2\)` or the **slope** associated with the age of the participant. -- - The key difference is that the predictions of the model can no longer be drawn as a line on a graph, but the interpretation of the parameters doesn't change that much, we just have to be careful on how we talk about them. -- - For example, `\(\beta_0\)` is the expected response time for a trial where the object is rotated by 0 degrees `\((x_{i1} = 0)\)` and the participant is 0 years old `\((x_{i2} = 0)\)`. -- - Typically, if the interpretation of the parameter doesn't make much sense in the context of a experiment (like a participant that is 0 years old), we just report the estimated value of the parameter. For example, "the estimate of the intercept was equal to..." --- ## Interpreting the slopes - The interpretation of the slopes also has to be adjusted. -- - For example, we now interpret `\(\beta_1\)` as the expected change on response time associated with a unit increase in the angle of rotation **while holding the age of participants constant**. -- - The main difference from our previous interpretation, is that this time we have to interpret it as the change in our dependent variable when the other independent variable, in this case **age**, are held constant. -- - If we allow both variables to change values at the same time, then we would not be able to interpret parameters as the change in our dependent variable, because the total change would be equal to the sum of both `\(\beta_1\)` and `\(\beta_2\)`. -- - Because of this, we always include the phrase "holding other variables constant" to our interpretation. This means that only the independent variable associated to the `\(\beta\)` value that we are looking at is allowed to change. --- ## Multiple linear regression - It is important to know that, whenever we add a new variable to a linear regression, the estimated values of the parameters that we had before will change. -- - In other words, the estimate `\(\hat{\beta}_1\)` from the simple linear model will not be the same as the estimate of `\(\hat{\beta}_1\)` in the multiple linear regression. -- - This is because some of the change in the dependent variable that was attributed only to our first independent variable `\((x_{i1})\)`, is now attributed to the second independent variable `\((x_{i12})\)`. -- - In other words, part of the change in a participants response time depends on the angle of rotation and some of that change depends on the age of the participant. --- ## Links for the final project data - Split Brain ```r link <- "https://raw.githubusercontent.com/ManuelVU/psych-10c-data/main/problem-1.csv" ``` - Cognitive Decline ```r link <- "https://raw.githubusercontent.com/ManuelVU/psych-10c-data/main/project-2.csv" ``` - Ikea Effect ```r link <- "https://raw.githubusercontent.com/ManuelVU/psych-10c-data/main/project-3.csv" ```